Rendering glasses shadows

ABSTRACT

Rendering glasses with shadows is disclosed, including: generating a face image corresponding to an image of a set of images based at least in part on a face model, wherein the set of images is associated with a user&#39;s face; generating a face with shadows image corresponding to the image based at least in part on shadows casted by a glasses model on the face model; generating a shadow transform based at least in part on a difference determined based at least in part on the face image and the face with shadows image; generating a shadowed image based at least in part on applying the shadow transform to the image; and presenting the shadowed image including by overlaying a glasses image associated with the glasses model over the shadowed image.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 14/603,207 entitled RENDERING GLASSES SHADOWS filed Jan. 22,2015 which is incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

Conventionally, in a virtual try-on of an item, such as a virtual pairof glasses displayed on a recorded image of a person's face, if theshadows casted by the glasses are not visible on the person's face, thenthe effect of the try-on is missing an important element of realism.Typically, a shadow on an image can be generated using ray tracing. Togenerate a shadow of an object (e.g., a virtual pair of glasses) on aperson's face, an exact model of the person's face, including a texturethat represents the person's face, may need to be generated before raytracing can be performed. However, not only is generating such adetailed model of the person's face time consuming, the process of raytracing a shadow onto the exact model is also very laborious andinefficient.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a diagram showing an embodiment of a system for renderingglasses shadows.

FIG. 2 is a diagram showing an embodiment of a server.

FIG. 3 is a flow diagram showing an embodiment of a process forrendering glasses shadows.

FIG. 4 is a flow diagram showing an example of a process for processinga set of images.

FIG. 5 is a diagram showing several images included in a set of images.

FIG. 6 is a flow diagram showing an example of a process for renderingglasses shadows.

FIG. 7 is a flow diagram showing an example of a process for determininga lighting model from a set of images of a user's face.

FIG. 8 is a flow diagram showing an embodiment of a process fordetermining a shadow transform corresponding to an image of a user'sface.

FIG. 9 is a diagram showing an example of an image of a user's face.

FIG. 10 is a diagram showing an example of a visualization of a 3Dgeneric face model in 3D space.

FIG. 11 is a diagram showing an example of a visualization of a 3D modelof a user's face in 3D space.

FIG. 12 is a diagram showing an example of a visualization of a morphed3D generic face model in 3D space.

FIG. 13 is a diagram showing an example of a visualization of atransformed 3D generic face model in 3D space.

FIG. 14 is a diagram showing an example of a 2D generic face image.

FIG. 15 is a diagram showing an example of a transformed 3D generic facemodel combined with a transformed 3D glasses model in 3D space.

FIG. 16 is a diagram showing an example of a 2D generic face withshadows image.

FIG. 17 is a diagram showing an example of a difference image determinedbased on a 2D generic face image and a 2D generic face with shadowsimage.

FIG. 18 is a diagram showing an example of an inverted difference image.

FIG. 19 is a diagram showing an example of an inverted difference imageto which a Despeckle filter has been applied.

FIG. 20 is a diagram showing an example of a previously processeddifference image to which a blur filter has been applied.

FIG. 21 is a diagram showing an example of a previously processeddifference image to which an RGB curves filter has been applied.

FIG. 22 is a diagram showing an example of a previously processeddifference image to which an HSV filter adjustment has been applied.

FIG. 23 is a diagram showing an example of an image of a user's face towhich a corresponding shadow transform has been applied.

FIG. 24 is a diagram showing an example presentation of a shadowed imagewith a 2D glasses image overlay.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Embodiments of rendering glasses shadows are described herein. Invarious embodiments, a three-dimensional (3D) representation (e.g.,model) of a user's face is determined from a set of images of the user'sface at various orientations. For example, the set of images may includeframes of a video recording of the user moving his or her head from sideto side. In various embodiments, a set of extrinsic information isdetermined for each image of the set of images. In some embodiments, theset of extrinsic information corresponding to an image describes anorientation associated with the user's face in that image. A 3D genericmodel of a face is obtained (e.g., from storage). In variousembodiments, the 3D generic model comprises a greyscale 3D model of ageneric user's face. In various embodiments, the 3D generic model ismorphed to match the 3D model of the user's face. In variousembodiments, for each image of the set of images associated with theuser's face, the morphed 3D generic model can be transformed with theset of extrinsic information associated with that particular image togenerate a two-dimensional (2D) face image corresponding to that image.In various embodiments, for each image of the set of images associatedwith the user's face, the morphed 3D generic model transformed with theset of extrinsic information associated with that particular image isfitted with a 3D glasses model that is also transformed with the set ofextrinsic information associated with that particular image and a shadowcasted by the transformed 3D glasses model onto the transformed generic3D model is simulated with a given lighting model. In some embodiments,the lighting model is determined based on the specific lightingenvironment of the user in the set of images. For example, the 3Dglasses model is associated with a pair of glasses that has beenselected to be virtually tried on the user's face. In variousembodiments, a 2D face with shadows image that includes the shadowscasted by the glasses model onto the transformed 3D generic model isgenerated. In various embodiments, a corresponding shadow transform isgenerated for each image of the set of images associated with the user'sface based at least in part on a difference between each image'scorresponding 2D face image and the corresponding 2D face with shadowsimage. In various embodiments, a shadow transform corresponding to animage of the set of images associated with the user's face can beapplied to that image to render/simulate the shadows casted by theselected glasses onto the user's face in that image. The image of theuser's face to which a corresponding shadow transform was applied can bepresented with an overlay of a 2D image of the selected pair of glassesso as to simulate a virtual try-on of the selected pair of glasses onthe user's face, with realistic shadows casted by the selected glassesonto the user's face as simulated using the lighting model that isdetermined based on the specific lighting environment of the user.

FIG. 1 is a diagram showing an embodiment of a system for renderingglasses shadows. In the example, system 100 includes client device 104,network 106, and server 108. Network 106 includes high speed datanetworks and/or telecommunications networks. In some embodiments, clientdevice 104 is configured to communicate to server 108 over network 106.

Client device 104 is configured to record or receive a set of recordedimages corresponding to a user's face/head at various orientations.Examples of client device 104 may include a laptop computer, a desktopcomputer, a tablet device, a mobile device, a smart phone and/or anycomputing device. For example, the set of recorded images may comprise avideo or a series of snapshots. In some embodiments, client device 104includes or is connected to a camera device. The camera device and/or aprocessor of client device 104 that is running an application cancapture a set of images of the user's face as user 102 turns his or herhead in different directions (e.g., as instructed through a userinterface of the application). In some embodiments, the set of images ofa user's face is captured/submitted at a user interface of a website oran application executing at client device 104. In various embodiments,the set of images is sent to server 108 for server 108 to process. Forexample, server 108 is associated with the website or applicationthrough which the set of images is captured. In some embodiments, clientdevice 104 includes a user interface through which the user may interactand view a playback associated with the images. For example, theplayback associated with the images may also be presented at the websiteor in the application.

In various embodiments, server 108 is configured to receive a set ofimages sent from a client device such as client device 104. Server 108searches for a (e.g., optimal) representation (e.g., a mathematical 3Dmodel) of the user's (e.g., user 102) face associated with a set ofimages and also (e.g., optimal) sets of extrinsic informationcorresponding to respective images of the set (e.g., a set of extrinsicinformation is specifically determined for each image of the set).

Server 108 is configured to obtain (e.g., from storage) a 3D genericface model. For example, the 3D generic face model was determined byaveraging previously generated 3D models of users' faces. In someembodiments, the 3D generic face model may be configured by anadministrator. In some embodiments, server 108 is configured to performa 3D reconstruction process to generate a 3D generic face model from aset of images. In some embodiments, the 3D generic face model comprisesa greyscale surface.

In various embodiments, server 108 is configured to morph the 3D genericface model to match the 3D model of the user 102's face. For example, inthe morphed 3D generic face model, at least certain portions of the 3Dgeneric face model have been moved to match corresponding portions ofthe 3D model of user 102's face in 3D space. Server 108 is configured touse the set of extrinsic information corresponding to each image toorient the morphed 3D generic face model to correspond to theorientation of the user's face in that image. Then, server 108 isconfigured to project the oriented morphed 3D generic face model onto a2D surface of the focal plane of a camera using a set of intrinsicinformation associated with the camera to obtain a 2D generic face imagecorresponding to that image of the set of images. As such, acorresponding 2D generic face image can be generated corresponding toeach image of the set of images.

Server 108 is configured to receive a selection of a pair of glasses.For example, the pair of glasses is selected to be tried onto the user'sface of the set of images. Server 108 obtains a 3D model of the selectedpair of glasses (e.g., from storage). Server 108 is configured to usethe set of extrinsic information corresponding to each image to orientthe 3D glasses model to correspond to the orientation of the user's facein that image. Server 108 is configured to fit a transformed 3D glassesmodel corresponding to an image onto the transformed 3D generic facemodel (that was transformed using the same set of extrinsic information)corresponding to the same image and apply a lighting model to thetransformed 3D generic face model fitted with the transformed 3D glassesmodel to obtain shadows casted by the transformed 3D glasses model ontothe transformed 3D generic face model. In some embodiments, the lightingmodel is determined based on the lighting environment of the user asdepicted in the set of images. Then, server 108 is configured to projectthe transformed 3D generic face model with the shadows casted by thetransformed 3D glasses model onto a 2D surface of the focal plane of acamera using a set of intrinsic information associated with the camerato obtain a 2D generic face with shadows image corresponding to thatimage of the set of images. As such, a corresponding 2D generic facewith shadows image can be generated corresponding to each image of theset of images.

In various embodiments, server 108 is configured to determine adifference image corresponding to each image of the set of imagescorresponding to a user's face/head between the 2D generic face withshadows image and the 2D generic face image corresponding to that image.As will be described in further detail below, each difference image canbe further processed and applied as a “shadow transform” to acorresponding image of the set of images of a user's face/head. Eachimage of the set of images corresponding to a user's face/head to whichthe shadow transform has been applied is referred to as a “shadowedimage” in various embodiments. Each “shadowed image” comprises theoriginal image of the user's face rendered with the corresponding shadowtransform processed from the 3D generic face model. Server 108 isconfigured to present at least some of the shadowed images correspondingto respective ones of the set of images corresponding to a user'sface/head at a user interface, including overlaying a corresponding 2Dimage of the selected pair of glasses over each shadowed image toemulate a virtual experience of trying on the selected pair of glassesfor the user, including presenting the rendered shadows as casted by theglasses in each displayed image to provide a realistic appearance of aglasses being worn on a face.

In some embodiments, enhancing the recorded set of images as describedabove may also be performed, at least in part, locally at client device104. For example, server 108 can send computer code to client device 104that client device 104 can use to perform at least a portion of theglasses shadows rendering as described herein.

FIG. 2 is a diagram showing an embodiment of a server. In someembodiments, server 108 of system 100 of FIG. 1 is implemented using theexample of FIG. 2. In the example, the server includes storage 202,shadow generator 204, model generator 206, extrinsic informationgenerator 208, intrinsic information generator 210, and rendering engine212. The server may be implemented with additional, different, and/orfewer components than those shown in the example. Each of shadowgenerator 204, model generator 206, extrinsic information generator 208,intrinsic information generator 210, and rendering engine 212 may beimplemented using hardware and/or software.

Storage 202 is configured to store data. In some embodiments, storage202 stores one or more sets of images and any associated data. Forexample, each set of images is associated with a video or a series ofsnapshots of various orientations of a user's face/head. In someembodiments, storage 202 stores one or more 3D generic face models. Insome embodiments, a 3D generic face model comprises a greyscale model ofa generic user face. In some embodiments, a 3D generic face model isgenerated based on the face models of previous users of the systemand/or a predetermined generic model.

In some embodiments, storage 202 stores one or more 3D glasses models.In some embodiments, each 3D glasses model is associated with a pair ofglasses for which a user can select to virtually try on. In someembodiments, storage 202 also stores glasses frame information of each3D glasses model. For example, the glasses frame information of each 3Dglasses model comprises the dimensions of the glasses, attributes of thelenses of the glasses (e.g., whether the lenses comprise a particularcolor), the manufacturer of the glasses, and the price of the glasses.In some embodiments, storage 202 also stores 2D images of each of one ormore 3D glasses models that have been oriented (transformed) tocorrespond to an orientation of a user's face in a set of images.

In some embodiments, storage 202 stores 2D generic face images of the 3Dgeneric face model that have been oriented (transformed) to correspondto the orientations of a user's face in a set of images. In someembodiments, storage 202 stores 2D generic face images with shadowsimages of a 3D generic face model that has been oriented (transformed)to correspond to the orientations of a user's face in a set of imagesand also have had shadows of a 3D glasses model casted onto the 3Dgeneric face model. In some embodiments, storage 202 stores a shadowtransform that corresponds to each image in a set of images of a user'sface that is determined based on a difference between a 2D generic faceimage corresponding to that image and a 2D generic face with shadowsimage corresponding to that image. In some embodiments, storage 202stores a shadowed image corresponding to each image in a set of imagesof a user's face that is determined based on applying the shadowtransform corresponding to the image to the image.

Model generator 206 is configured to determine a mathematical 3D modelfor a user's face associated with each set of images of a user's face.For example, the mathematical 3D model of the user's face (i.e., themathematical model of the user's face in 3D space) may be set at theorigin. In some embodiments, the mathematical 3D model determined for auser's face is referred to as an M matrix. In some embodiments, the Mmatrix may be determined based on a set of reference points associatedwith features on the user's face from the associated set of images.Examples of reference points include endpoints of the user's eye, bridgeof the user's nose, and tip of the user's nose. In some embodiments,model generator 206 is configured to store the M matrix determined for aset of images with the set at storage 202.

Extrinsic information generator 208 is configured to determine a set ofextrinsic information for each of at least a subset of a set of imagesof a user's face. For example, the set of images may be stored atstorage 202. In various embodiments, a set of extrinsic informationcorresponding to an image of a set of images describes one of more ofthe orientation, rotation, and translation of the 3D model of the user'sface needed to result in the correct appearance of the user's face inthat particular image. In some embodiments, the set of extrinsicinformation determined for an image of a set of images associated with auser's face is referred to as an (R, t) pair where R is a rotationmatrix and t is a translation vector corresponding to that image. Insome embodiments, extrinsic information generator 208 is configured tostore the (R, t) pair determined for each of at least a subset of a setof images with the set at storage 202.

Intrinsic information generator 210 is configured to generate a set ofintrinsic information for a camera associated with recording a set ofimages of a user's face. For example, the camera was used to record aset of images stored at storage 202. In various embodiments, a set ofintrinsic information corresponding to a camera describes a set ofparameters associated with the camera. For example, a parameterassociated with a camera comprises a focal length and a principal pointof the camera sensor. In some embodiments, the set of intrinsicinformation associated with a camera is found by taking multiplepictures under various angles of an object with known points and solvingan optimization problem treating camera intrinsics and extrinsics asvariables, and the model as a constant. In some embodiments, the set ofintrinsic information associated with a camera is referred to as an Imatrix. In some embodiments, for many practical uses, includingrendering glasses shadows onto images made on laptops or mobile phones,the intrinsic information determined for the camera can be assumed to bethe same across all devices. In some embodiments, intrinsic informationgenerator 210 is configured to store an I matrix determined for thecamera associated with a set of images with the set at storage 202.

Shadow generator 204 is configured to morph a 3D generic face model tomatch a 3D model of a user's face determined from a set of images of auser's face. In some embodiments, the modified 3D generic face modelcorresponding to the user's face is referred to as the M′ matrix. Insome embodiments, the shadow generator 204 morphs the 3D generic facemodel by moving certain points of the 3D generic face model to matchcorresponding locations on the 3D model of the user's face. Shadowgenerator 204 is configured to generate a shadow transform correspondingto each image from a set of images of a user's face based on firstgenerating at least two 2D images from the modified 3D generic facemodel that was morphed to match the 3D model of the user's face. Invarious embodiments, to generate a first image of these at least twoimages corresponding to an image of the user's face (e.g., stored atstorage 202), shadow generator 204 is configured to transform themodified 3D generic face model using an (R, t) pair (extrinsicinformation) corresponding to that image and project the transformed 3Dgeneric face model onto the 2D surface of the focal plane of the camerato obtain the 2D generic face image corresponding to that image. Inother words, I×(R×M′+t) results in the projection of the modified 3Dgeneric face model, the M′ matrix, in the orientation and translationtransformed by the (R, t) pair corresponding to an image of a user'sface, onto a 2D surface. The projection onto the 2D surface is the viewof the transformed 3D generic face model as seen from the camera. Insome embodiments, a 2D generic face image comprises a greyscale image.In some embodiments, the 2D generic face image comprises an image file(e.g., a .png, a .jpeg, etc.). In some embodiments, the 2D generic faceimage comprises a matrix with greyscale pixel values corresponding tothe pixels of the 2D image. In various embodiments, to generate a secondimage of these at least two images corresponding to the same image ofthe user's face (e.g., stored at storage 202), shadow generator 204 isconfigured to transform a 3D model of a selected pair of glasses usingthe (R, t) pair (extrinsic information) corresponding to that image andcombine the transformed 3D glasses model with the transformed 3D genericface model. In some embodiments, combining the transformed 3D glassesmodel with the transformed 3D generic face model comprisesplacing/fitting the transformed 3D glasses model onto the transformed 3Dgeneric face model. Shadow generator 204 is then configured to apply alighting model to the combination of the transformed 3D generic facemodel and the transformed 3D glasses model to create the shadows castedby the transformed 3D glasses model onto the transformed 3D generic facemodel. Shadow generator 204 then projects the transformed 3D genericface model with the shadows casted by the 3D glasses model onto the 2Dsurface of the focal plane of the camera to obtain the 2D generic facewith shadows image corresponding to that image of the user's face. Theprojection onto the 2D surface is the view of the transformed 3D genericface model with the shadows of the transformed 3D glasses model as seenfrom the camera. In some embodiments, a 2D face with shadows imagecomprises a greyscale image or non-greyscale image. In some embodiments,the 2D face with shadows image comprises an image file (e.g., a .png, a.jpeg, etc.). In some embodiments, the 2D generic face with shadowsimage comprises a matrix with greyscale pixel values corresponding tothe pixels of the 2D image. In various embodiments, shadow generator 204is configured to determine the shadow transform for the image of theuser's face based at least in part on the difference between the secondimage, the 2D generic face with shadows image, and the first image, the2D generic face image. In some embodiments, a shadow transform comprisesa greyscale or non-greyscale image. In some embodiments, the shadowtransform comprises an image file (e.g., a .png, a .jpeg, etc.).

Shadow generator 204 is configured to apply a determined shadowtransform to each image of the at least subset of the set of images ofthe user's face to generate a corresponding shadowed image. Shadowgenerator 204 is configured to apply a shadow transform to each(original) image of the user's face by multiplying the original image ofthe user's face by the shadow transform. For example, modifying eachoriginal image using the corresponding shadow transform includesmultiplying each channel (e.g., Red, Green, and Blue) value of eachpixel of the original image with the corresponding value from thecorresponding pixel of the shadow transform.

Rendering engine 212 is configured to display at least some shadowedimages at a user interface. In some embodiments, rendering engine 212 isconfigured to display a corresponding 2D image of the selected pair ofglasses overlaid over each of at least some of the shadowed images toprovide the user the experience of virtually trying on the selected pairof glasses and with realistic shadows casted onto the user's face by theselected pair of glasses in each played back image.

FIG. 3 is a flow diagram showing an embodiment of a process forrendering glasses shadows. In some embodiments, process 300 isimplemented at system 100 of FIG. 1.

At 302, a face image corresponding to an image of a set of images isgenerated based at least in part on a face model, wherein the set ofimages is associated with a user's face. A first 2D image is generatedcorresponding to an image of the user's face based on a 3D generic facemodel. In some embodiments, the 3D generic face model has been morphedto match a 3D model of the user's face that was determined from the setof images of the user's face. In some embodiments, a corresponding setof extrinsic information that describes the orientation of the user'sface in that particular image is determined. In various embodiments,prior to generating the 2D generic face image from the modified 3Dgeneric face model, the modified 3D generic face model is transformed bythe set of extrinsic information corresponding to that particular image.

At 304, a face with shadows image corresponding to the image isgenerated based at least in part on shadows casted by a glasses model onthe face model. A second 2D image is generated corresponding to theimage of the user's face based on the shadows casted by a 3D glassesmodel onto the 3D generic face model. In some embodiments, prior togenerating the 2D generic face with shadows image from the modified 3Dgeneric face model, the modified 3D generic face model is transformed bythe set of extrinsic information corresponding to the image and alsocombined with a 3D glasses model (e.g., corresponding to a selected pairof glasses). A lighting model is applied to the combination of thetransformed 3D generic face model and the transformed 3D glasses modelto simulate the shadows casted by the 3D glasses model onto the 3Dgeneric face model. In various embodiments, the lighting model emulatesthe light that is shone onto the user's face in the set of images. Insome embodiments, the lighting model is derived from the set of images.For example, the casted shadows may include the shadows caused by thelenses of the glasses and/or the frame of the glasses.

At 306, a shadow transform is generated based at least in part on adifference determined based at least in part on the face image and theface with shadows image. The shadow transform corresponding to the imageof the user's face is determined based at least in part on thedifference image determined by subtracting the first image, the faceimage, from the second image, the face with shadows image. In someembodiments, the difference image is further processed (e.g., with oneor more filters) before it is used as the shadow transform correspondingto the image of the user's face.

At 308, a shadowed image is generated based at least in part on applyingthe shadow transform to the image. In various embodiments, applying theshadow transform to the image of the user's face includes multiplyingthe image of the user's face by the shadow transform. The shadowed imagecomprises the original image of the user's face with the addition of theshadows that were casted by the 3D glasses model onto the modified 3Dgeneric face model that had been transformed to match the orientation ofthe user's face in that original image.

At 310, the shadowed image is presented including by overlaying aglasses image associated with the glasses model over the shadowed image.The shadowed image is overlaid with a 2D image of the glasses prior tobeing displayed. In some embodiments, the 2D image of the glasses wasdetermined from projecting the 3D glasses model transformed by the setof extrinsic information corresponding to the image of the user's faceonto a 2D surface of the focal plane of a camera using a set ofintrinsic information associated with the camera.

FIG. 4 is a flow diagram showing an example of a process for processinga set of images. In some embodiments, process 400 is implemented atsystem 100 of FIG. 1.

At 402, a recorded set of images is received. In various embodiments,the set of images corresponds to a recorded video or a series ofsnapshots of a user's face turned in different orientations. As such,each image of the set is associated with an orientation of the user'sface in that image.

At 404, a representation of a user's face associated with the set ofimages and a plurality of sets of extrinsic information corresponding torespective ones of at least a subset of the set of images are searchedfor. In some embodiments, the representation of the user's face is amodel in 3D space and is referred to as an M matrix. In someembodiments, the M matrix is determined based at least in part onmeasured reference points of one or more features associated with theuser's face. In some embodiments, the extrinsic set of informationassociated with an image of the set of images is referred to as an (R,t) pair. An (R, t) pair is determined for each of at least a subset ofthe set of images so each image corresponds to a respective (R, t) pairthat is associated with the orientation of the user's face in thatimage.

In some embodiments, an optimal M matrix is determined for the set ofimages and an optimal (R, t) pair is determined for each of at least asubset of the set of images. In a first example, a parameter search isused to perform iterative computations until the optimal M and set of(R, t) pairs are found. For example, a distribution of M matrices (e.g.,that have been predetermined based on known face samples or generated onthe fly) corresponding to the set of images and a distribution of (R, t)pairs corresponding to each image of the set of images are determined,and a combination of matrix M and (R, t) pairs that best describes atleast a subset of the set of images is selected. In another example, abundle adjustment technique is used and the bundle adjustment techniquemay treat the M and the set of (R, t) pairs as unknowns in anoptimization problem and iteratively test out various combinations of Mmatrices and (R, t) pairs until an M and a set of (R, t) pairs are foundthat best match the set of images. For example, the optimal M matrix andan optimal (R, t) pair corresponding to an image result in the minimumreprojection error of any other combination of an M matrix and an (R, t)pair and therefore the combination of this M matrix and this (R, t) pairbest matches the image corresponding to the (R, t) pair. While one Mmatrix is determined for the set of images, a set of (R, t) pairs, eachcorresponding to respective ones of at least a subset of the set ofimages, is determined.

FIG. 5 is a diagram showing several images included in a set of images.In the example, Images 1, 2, 3, 4, and 5 are included in a recorded setof images of a user's face at various different orientations relative tothe camera. As described above, a set of extrinsic information, E,(e.g., an (R, t) pair) that describes the orientation and translation ofthe user's face in an individual image is determined for that image. Asshown in the example, each image is identified by a number 1 through 5and has a corresponding set of extrinsic information, E, which includesthe number associated with the image as its subscript (e.g., E₁corresponds to Image 1, E₂ pair corresponds to Image 2, etc.). Forexample, E₁ may be used to transform a 3D generic face model that hasbeen morphed to match a 3D model of the user's face in the set ofimages, the M matrix, in 3D space into the orientation and translationof the user's face that is shown in Image 1.

FIG. 6 is a flow diagram showing an example of a process for renderingglasses shadows. In some embodiments, process 600 is implemented atsystem 100 of FIG. 1. In some embodiments, process 300 of FIG. 3 isimplemented by a process such as process 600.

At 602, a modified 3D generic face model is determined by modifying a 3Dgeneric face model to match a 3D model of a user's face. In someembodiments, a 3D generic face model comprises a greyscale model of ageneric face. In some embodiments, a 3D generic face model comprises aBlender file. In some embodiments, a 3D generic face model can compriseany 3D model file format that is supported by a rendering engine. Insome embodiments, modifying the 3D generic face model includes morphingthe 3D generic face model to change its shape to correspond to the shapeof the 3D model of the user's face. For example, the 3D generic facemodel is morphed to match a 3D model of a user's face (e.g., that wasdetermined using a process such as process 400 of FIG. 4) by at leastmoving certain locations of the 3D generic face model to matchcorresponding locations on the 3D model of the user's face.

The following is one example technique by which to morph the 3D genericface model to correspond to the representation of the user's face:Define a set of reference points on a face. Examples of reference pointsmay be related to facial features such as eye corners, a nose tip,cheekbones, and mouth corners. The 3D coordinate of each such referencepoint is located on the representation (e.g., 3D model) of the user'sface and also on the 3D generic face model. The 3D generic face model ismorphed to correspond to the 3D model of the user's face by moving theeach reference point located on the 3D generic face model to match the3D coordinate of the corresponding reference point located on the 3Dmodel of the user's face. As each reference point located on the 3Dgeneric face model is moved to match the 3D coordinate of thecorresponding reference point located on the 3D model of the user'sface, the neighboring portions of the 3D generic face model are pulledalong with the movement.

At 604, for an image of the plurality of images associated with theuser's face, a set of extrinsic information corresponding to the imageand the modified 3D generic face model are used to generate atransformed 3D generic face model. A set of extrinsic information isdetermined corresponding to each image of the plurality of images (e.g.,using a process such as process 400 of FIG. 4). The modified 3D genericface model is transformed by the set of extrinsic information determinedfor the image to orient the modified 3D generic face model to match theorientation of the user's face in the image.

At 606, a 2D generic face image corresponding to the image is generatedbased at least in part on the transformed 3D generic face model. A 2Dgeneric face image corresponding to the image is generated by projectingthe transformed 3D generic face model onto a 2D surface of the focalplane of a camera using a set of intrinsic information associated withthe camera. For example, the 2D generic face image comprises a 2Dprojection of the transformed 3D generic face model that corresponds tothe orientation of the user's face in that image.

At 608, the set of extrinsic information and a 3D glasses model are usedto generate a transformed 3D glasses model. In various embodiments, apair of glasses is selected to be virtually tried on by a user (e.g.,associated with the plurality of images). In some embodiments, the pairof glasses is selected by a user. In some embodiments, the pair ofglasses is selected based on a determined fit score between the glassesand the user associated with the plurality of images. In someembodiments, the fit score of a pair of glasses with respect to a useris determined by comparing one or more head measurements (e.g., the headmeasurements are determined based on reference points includingendpoints of the user's eyebrow, endpoints of the user's eye, bridge ofthe user's nose, tip of the user's nose, etc.) determined from theplurality of images of the user's face against the stored information(e.g., dimensions of the glasses including a bridge length, a lensdiameter, a temple distance, etc.) associated with various pairs ofglasses. For example, a penalty function is used to evaluate thecomparison between the user's head measurements and the glassesinformation to determine a fit score for the glasses. Various differentglasses may be ranked based on their respective fit scores and a pair ofglasses can be selected based on their corresponding fit score. Forexample, a pair of glasses with the highest fit score can beautomatically selected to be tried on by the user.

A stored 3D model of the selected pair of glasses is also transformed bythe set of extrinsic information determined for the image to orient the3D glasses model to match the orientation of the user's face in theimage.

At 610, a combination of the transformed 3D generic face model and thetransformed 3D glasses model is generated. The transformed 3D glassesmodel is placed on the transformed 3D generic face model such that theglasses model covers the eyes of the generic face model. The followingis one example technique by which to place the 3D glasses model over thetransformed 3D generic face: For each 3D model of a user's face, where a3D model of a generic pair of glasses should be placed relative to theuser's face is determined. A 3D model of a generic pair of glassescomprises a set of 3D points, including, for example: two bridge pointsand two temple points. For example, a heuristic algorithm that receiveseye corners, the nose tip, and ear junctions of the 3D model of theuser's face as inputs can be used to compute the locations on the 3Dmodel of the user's face that match the two bridge points and two templepoints of the 3D model of the generic pair of glasses. Once theplacement of the 3D model of the generic pair of glasses on the 3D modelof the user's face is determined, for each 3D model of a specific pairof glasses, a transformation is computed to place the 3D model of thespecific pair of glasses on the 3D model of the user's face such thatthe distance between the points of the 3D model of the specific pair ofglasses and the points of the 3D model of the generic pair of glasses isminimized.

At 612, a lighting model is applied to the combination to generateshadows casted on the transformed 3D generic face model by thetransformed 3D glasses model. A lighting model is applied to thetransformed 3D generic face model wearing the transformed 3D glassesmodel to generate the shadows casted by the glasses model onto thegeneric face model. For example, the shadows can be generated on thegeneric face model using ray-tracing. In some embodiments, the lightingmodel emulates the lighting directed at the user's face in the specificlight environment in which the user is located. In some embodiments, thelighting model is determined from the plurality of images of the user'sface. FIG. 7, below, describes an example process of determining such alighting model.

At 614, a 2D generic face with shadows image corresponding to the imageis generated based at least in part on the shadows casted on thetransformed 3D generic face model by the transformed 3D glasses model. A2D generic face with shadows image corresponding to the image isgenerated by projecting the transformed 3D generic face model with theshadows casted by the transformed 3D glasses model onto a 2D surface ofthe focal plane of a camera using a set of intrinsic informationassociated with the camera. In various embodiments, the shadows castedby the frame of the glasses as well as the shadows casted by the lensesof the glasses are rendered on the generic face model. For example, ifthe lenses were colored, then the casted shadows would include suchcorresponding color. In various embodiments, the glasses model is notincluded in the 2D generic face with shadows image and instead, only theshadows that the glasses model had casted is included in the 2D genericface with shadows image.

At 616, a shadow transform corresponding to the image is generated basedat least in part on a difference between the 2D generic face image andthe 2D generic face with shadows image. A shadow transform correspondingto the image comprises a difference image generated by subtracting the2D generic face image (generated at step 606) from the 2D generic facewith shadows image (generated at step 614). The difference imagecomprises a 2D image of only the shadows casted by the glasses modelonto the generic face. As mentioned above, the shadows may be ingreyscale or in color, depending on the color of the glasses frameand/or glasses lens. In some embodiments, the difference image isfurther processed before being used as the shadow transform. FIG. 8,below, describes an example process of generating a shadow transformincluding by further processing the difference image.

At 618, a shadowed image corresponding to the image is generated basedat least in part on applying the shadow transform to the image. Invarious embodiments, a shadowed image corresponding to the image appearslike the original image with the addition of the shadow transform in theareas of the face (e.g., the eyes) that were affected by the shadowscasted by the glasses model. In various embodiments, a shadowed imagecorresponding to the image is generated by multiplying the originalimage with the corresponding shadow transform. For example, if the pixelvalues of the shadow transform were not on a scale of 0 to 1, they arescaled to be on a scale of 0 to 1. Then, the scaled pixel values of theshadow transform are multiplied with each channel (e.g., Red, Green, andBlue) value of each corresponding pixel of the original image. Putanother way, the shadow transform can be thought of as being placed overthe original image and a channel value of each pixel of the shadowedimage is determined by multiplying the channel value of each pixel ofthe original image with a corresponding value of a corresponding pixelof the shadow transform.

At 620, it is determined whether there is at least one more image in theplurality of images. In the event that there is at least one more imagein the plurality of images, control is transferred to step 622, at whichthe next image in the plurality of images is to be addressed startingagain from step 604. Otherwise, in the event that there is not at leastone more image in the plurality of images, control is transferred tostep 624.

At 624, one or more shadowed images overlaid with respective 2D glassesimages associated with the 3D glasses model are presented. Each shadowedimage corresponding to at least a subset of the plurality of images ispresented with an overlay of a 2D image of the selected pair of glasses.For example, the 2D image of the selected pair of glasses can begenerated for each image by transforming the 3D glasses model using theset of extrinsic information associated with that image and thenprojecting the transformed 3D glasses model onto a 2D surface of thefocal plane of a camera using a set of intrinsic information associatedwith the camera. While each shadowed image shows only the simulatedshadows as casted by the glasses, the overlay of the corresponding 2Dglasses image completes the effect of glasses being virtually worn bythe user and therefore casting such shadows. Each presented shadowedimage is presented with a corresponding 2D glasses image. For example, auser may interact with this playback of shadowed images to skip throughdifferent views of the user's face with the rendered glasses image andcorresponding glasses shadows casted on the user's face.

As described above, because the shadow transform was rendered using a 3Dgeneric face model and then applied to an actual image of a user's face,the glasses shadows effect can be efficiently produced much faster thanwould be possible if a detailed 3D model of the user's face (e.g., a 3Dmodel with a texture of the user's skin) was first generated and thenthe glasses shadows were produced on the detailed 3D model of the user'sface.

FIG. 7 is a flow diagram showing an example of a process for determininga lighting model from a set of images of a user's face. In someembodiments, process 700 is implemented at system 100 of FIG. 1.

For example, the lighting model used in step 612 of process 600 of FIG.6 is determined using a process such as process 700. Process 700describes an example process of determining a model that emulates thelighting received in the set of images of a user's face.

At 702, for an image of a plurality of images associated with a user'sface, an area of the user's face in the image is identified. The areawithin an image of the user's face and/or head that contains the user'sface is detected and identified.

At 704, the identified area is divided into a predetermined set ofsegments. In some embodiments, the identified area is divided into oneor more segments, each of a predetermined shape and location relative tothe identified area of the user's face in the image. For example, theidentified area of the user's face in the image can be divided into fivesegments (e.g., rectangular segments) along the front of the user'sface/head, one segment (e.g., a polygonal segment) on top of the user'sface/head, one segment under the user's face/head, and one segment alongthe back of the user's face/head.

At 706, a set of lighting parameters associated with each of thepredetermined set of segments of the identified area is determined. Aset of lighting parameters is determined for each segment of theidentified area of the user's face/head in the image. Examples oflighting parameters include an intensity of the light and a color of thelight that are received in each segment in that image. Other examples oflighting parameters to be determined for each segment may include adirection of the light and a source of the light. In some embodiments,in addition to the set of lighting parameters associated with eachsegment, ambient lighting parameters are determined from the image. Forexample, ambient lighting parameters describe light that has no origin,no direction, and has an equal effect on all objects in the scene.

At 708, it is determined whether there is at least one more image in theset. In the event that there is at least one more image in the pluralityof images, control is transferred to step 710, at which the next imagein the plurality of images is to be addressed starting again from step702. Otherwise, in the event that there is not at least one more imagein the plurality of images, control is transferred to step 712.

At 712, a lighting model is determined based at least in part on thesets of lighting parameters determined from the plurality of images. Thesets of lighting parameters determined for respective ones of images inthe plurality of images are used together to determine a lighting model.For example, the average of the lighting parameters (e.g., the averagedirection of the light, the average source of the light, the averageintensity of the light, and/or the average color of the light) receivedin the same segment across all the images is determined to represent thelighting received by and therefore emitted by that segment of thelighting model.

For example, a lighting model comprises a 3D container that is formed bythe predetermined segments and where each segment emits light inside thecontainer based on the computed lighting received by the segment inconstructing the lighting model. To apply the lighting model to thetransformed 3D generic face model wearing the transformed 3D glassesmodel, the transformed 3D generic face model wearing the transformed 3Dglasses model is placed inside the 3D container associated with thelighting model and the computed light is emitted from each segment ontothe transformed 3D glasses model to cause the glasses model to castshadows onto the transformed 3D generic face model.

FIG. 8 is a flow diagram showing an embodiment of a process fordetermining a shadow transform corresponding to an image of a user'sface. In some embodiments, process 800 is implemented at system 100 ofFIG. 1. In some embodiments, step 616 of process 600 of FIG. 6 isimplemented with process 800.

At 802, a difference image is generated by subtracting a 2D generic faceimage from a 2D generic face with shadows image. A 2D generic face imagecorresponding to an image of a user's face and a 2D generic face withshadows image corresponding to the image of the user's face are alreadygenerated (e.g., using steps such as 602 through 614 of process 600 ofFIG. 6). The resulting image, the difference image, when the 2D genericface image is subtracted from the 2D generic face with shadows image,comprises a 2D image of only the shadows casted by the transformed 3Dglasses model onto the transformed 3D generic face model. The shadowsmay be casted by the frame of the glasses and/or the lenses of theglasses. In the event that the lenses of the glasses are colored, thenthe difference image includes the color.

At 804, colors associated with the difference image are inverted. Thedifference image is inverted by inverting all the pixel colors andbrightness values in the image. Due to inversion, the dark areas willbecome bright and the bright areas will become dark. Furthermore, huesare replaced by their complementary colors.

At 806, optionally, a Despeckle filter is applied to the inverteddifference image. This step of processing the inverted difference imageis optionally performed. Applying the Despeckle filter to the differenceimage locates and removes small clusters or other forms of renderingnoise. The Despeckle filter replaces each pixel with the median value ofthe pixels within a specified radius of pixels.

At 808, optionally, a blur filter is applied to the inverted differenceimage. This step of processing the inverted difference image isoptionally performed. In some embodiments, if this step is performed, itis performed after application of the Despeckle filter to the inverteddifference image. Applying a blur filter (e.g., a Gaussian filter) blursand/or softens the image. For example, blurring the image with aGaussian kernel includes computing a new pixel value for each pixel ofthe image based on a weighted average of the pixel values in thatpixel's neighborhood. The original pixel's value receives the heaviestweight (e.g., having the highest Gaussian value) and neighboring pixelsreceive smaller weights as their distance to the original pixelincreases.

At 810, optionally, a red, green, and blue (RGB) curves filter isapplied to the inverted difference image. This step of processing theinverted difference image is optionally performed. In some embodiments,if this step is performed, it is performed after application of the blurfilter to the inverted difference image. Applying an RGB curves filteradjusts the contrast in the image. For example, applying the RGB curvesfilter allows a user to individually adjust the quantity of each of theRed, Green, and Blue channels to change the contrast of the differenceimage.

At 812, optionally, a hue, saturation, and value (HSV) filter adjustmentis applied to the inverted difference image. This step of processing theinverted difference image is optionally performed. In some embodiments,if this step is performed, it is performed after application of the RGBcurves filter to the inverted difference image. Applying an HSV filtercauses the color of the lenses of the glasses in the image to be lesssubtle. In some embodiments, predetermined settings have been set forthe HSV filter. As such, the HSV filter can be applied without userinteraction. For example, the predetermined HSV filter parameters aredetermined based on manual verification of a selected set of 3D modelsof glasses.

At 814, the processed inverted difference image is determined tocomprise a shadow transform. The inverted difference image that has beenprocessed by one or more processors is used as a shadow transform forthe image of the user's face.

FIGS. 9 through 24 below show example visualizations associated withvarious steps of rendering glasses shadows for one image of a user'sface based on the embodiments described above.

FIG. 9 is a diagram showing an example of an image of a user's face.User image 900 can be one of multiple images that are associated withthe user's face/head at various angles. User image 900 comprises a 2Dimage of the user's face as the user is front facing the camera. Forexample, user image 900 comprises a photograph or a frame from a video.A set of extrinsic information that describes the orientation (e.g.,rotation and translation of a 3D model of the user's face to match theuser's face in user image 900) of the user's face in user image 900 isalready determined. FIGS. 10 through 24 below will describe an exampleof generating a shadow transform corresponding to user image 900 and atechnique of using the shadow transform.

FIG. 10 is a diagram showing an example of a visualization of a 3Dgeneric face model in 3D space. For example, 3D generic face model 1000was stored. For example, 3D generic face model 1000 is modeled from the3D models of previous users of the system and/or is determined on thefly. In various embodiments, 3D generic face model 1000 comprises agreyscale model. For example, 3D generic face model 1000 is a Blenderfile.

FIG. 11 is a diagram showing an example of a visualization of a 3D modelof a user's face in 3D space. For example, 3D model of a user's face1100 was constructed using a set of images of a user's face thatincludes user image 900 of FIG. 9. For example, 3D model of a user'sface 1100 was determined based on a set of reference points associatedwith features on the user's face from the set of images. Examples ofreference points include endpoints of the user's eye, bridge of theuser's nose, and tip of the user's nose. In some embodiments, 3D modelof a user's face 1100 was determined using a process such as process 400of FIG. 4.

FIG. 12 is a diagram showing an example of a visualization of a morphed3D generic face model in 3D space. In some embodiments, a 3D genericface model is morphed to match a 3D model of a user's face. In theexample of FIG. 12, morphed 3D generic face model 1200 was the result ofmorphing 3D generic face model 1000 of FIG. 10 to match 3D model of auser's face 1100 of FIG. 11. Morphing 3D generic face model 1000 of FIG.10 to match 3D model of a user's face 1100 of FIG. 11 includes, forexample, moving the certain areas of 3D generic face model 1000 of FIG.10 to corresponding locations on 3D model of a user's face 1100 of FIG.11 in 3D space. As such, morphed 3D generic face model 1200 has asimilar shape and features as those of 3D model of a user's face 1100 ofFIG. 11. For example, morphed 3D generic face model 1200 is an exampleof a morphed 3D generic face model that is determined in step 602 ofprocess 600 of FIG. 6.

FIG. 13 is a diagram showing an example of a visualization of atransformed 3D generic face model in 3D space. Transformed 3D genericface model 1300 comprises morphed 3D generic face model 1200 of FIG. 12after it was transformed to match the orientation of user image 900 ofFIG. 9 using the set of extrinsic information corresponding to userimage 900 of FIG. 9. As such, transformed 3D generic face model 1300 isalso front facing the camera. For example, transformed 3D generic facemodel 1300 is an example of a transformed 3D generic face model that isgenerated in step 604 of process 600 of FIG. 6.

FIG. 14 is a diagram showing an example of a 2D generic face image. 2Dgeneric face image 1400 comprises a projection of transformed 3D genericface model 1300 of FIG. 13 onto a 2D surface of the focal plane of acamera using a set of intrinsic information associated with the camera.2D generic face image 1400 comprises a rendering using primitivematerial and a low number of rays in a ray-tracing engine. For example,2D generic face image 1400 is an example of a 2D generic face image thatis generated in step 606 of process 600 of FIG. 6.

FIG. 15 is a diagram showing an example of a transformed 3D generic facemodel combined with a transformed 3D glasses model in 3D space.Transformed 3D generic face model wearing a transformed 3D glasses model1500 comprises transformed 3D generic face model 1300 of FIG. 13 fittedwith 3D glasses model 1502 that was transformed to match the orientationof transformed 3D generic face model 1300 of FIG. 13. For example, 3Dglasses model 1502 is associated with a pair of glasses that wasselected to be virtually tried on by the user. For example, 3D glassesmodel 1502 was also transformed using the set of extrinsic informationcorresponding to user image 900 of FIG. 9. As such, the transformedglasses model matches the front facing orientation of transformed 3Dgeneric face model 1300 of FIG. 13 and is also fitted onto transformed3D generic face model 1300 of FIG. 13 such that transformed 3D genericface model 1300 of FIG. 13 appears to be wearing 3D glasses model 1502.For example, transformed 3D generic face model wearing a transformed 3Dglasses model 1500 is an example of a combination of the transformed 3Dgeneric face model with the transformed 3D glasses model that isgenerated in step 610 of process 600 of FIG. 6.

A lighting model can then be applied to transformed 3D generic facemodel wearing a transformed 3D glasses model 1500 to generate theshadows casted by the 3D glasses model onto the 3D generic face model(not shown in FIG. 15).

FIG. 16 is a diagram showing an example of a 2D generic face withshadows image. 2D generic face with shadows image 1600 comprises aprojection of transformed 3D generic face model wearing a transformed 3Dglasses model 1500 of FIG. 15, after the glasses shadows were generated,onto a 2D surface of the focal plane of a camera using a set ofintrinsic information. 2D generic face with shadows image 1600 comprisesa 2D image of the generic face with the shadows casted by the glassesmodel but does not include the glasses themselves. 2D generic face withshadows image 1600 comprises a rendering using primitive material and alow number of rays in a ray-tracing engine. For example, 2D generic facewith shadows image 1600 is an example of a 2D generic face with shadowsimage that is generated in step 614 of process 600 of FIG. 6.

FIG. 17 is a diagram showing an example of a difference image determinedbased on a 2D generic face image and a 2D generic face with shadowsimage. Difference image 1700 is determined by subtracting 2D genericface image 1400 of FIG. 14 from 2D generic face with shadows image 1600of FIG. 16. Difference image 1700 may include only greyscale colors ornon-grayscale colors, depending on the color of the glasses model'sframe and lenses. For example, difference image 1700 comprises thedifference image generated at step 802 of FIG. 8.

FIG. 18 is a diagram showing an example of an inverted difference image.Inverted difference image 1800 comprises the inversion of differenceimage 1700 of FIG. 17. As such, the dark and light colors of differenceimage 1700 of FIG. 17 are inverted in inverted difference image 1800.For example, inverted different image 1800 is the result of step 804 ofprocess 800 of FIG. 8

FIG. 19 is a diagram showing an example of an inverted difference imageto which a Despeckle filter has been applied. Processed inverteddifference image 1900 comprises inverted difference image 1800 of FIG.18 after a Despeckle filter has been applied to it. The Despeckle filterhas removed some of the rendering noise from inverted difference image1800 of FIG. 18. For example, processed inverted difference image 1900comprises the result of step 806 of process 800 of FIG. 8.

FIG. 20 is a diagram showing an example of a previously processeddifference image to which a blur filter has been applied. Processedinverted difference image 2000 comprises processed difference image 1900of FIG. 19 after a blur filter has been applied to it. The blur filterhas smoothed processed difference image 1900 of FIG. 19. An example blurfilter is a Gaussian filter. For example, processed inverted differenceimage 2000 comprises the result of step 808 of process 800 of FIG. 8.

FIG. 21 is a diagram showing an example of a previously processeddifference image to which an RGB curves filter has been applied.Processed inverted difference image 2100 comprises processed differenceimage 2000 of FIG. 20 after an RGB curves filter has been applied to it.The RGB curves filter has adjusted the image contrast of processeddifference image 2000 of FIG. 20. In the example of FIG. 21, the RGBcurves filter has increased the image contrast of processed differenceimage 2000 of FIG. 20 to generate processed inverted difference image2100. For example, processed inverted difference image 2100 comprisesthe result of step 810 of process 800 of FIG. 8.

FIG. 22 is a diagram showing an example of a previously processeddifference image to which an HSV filter adjustment has been applied.Processed inverted difference image 2200 comprises processed differenceimage 2100 of FIG. 21 after an HSV filter adjustment has been applied toit. The HSV filter has emphasized the color of the lenses of the glassesin processed difference image 2100 of FIG. 21. For example, processedinverted difference image 2200 comprises the result of step 812 ofprocess 800 of FIG. 8.

In some embodiments, processed inverted difference image 2200 is alsoconsidered as the shadow transform corresponding to user image 900 ofFIG. 9.

FIG. 23 is a diagram showing an example of an image of a user's face towhich a corresponding shadow transform has been applied. An image towhich a corresponding shadow transform has been applied is alsosometimes referred to as a shadowed image. Shadowed image 2300 isgenerated by multiplying user image 900 of FIG. 9 with its correspondingshadow transform, processed inverted difference image 2200 of FIG. 22.As shown in the example, shadowed image 2300 still preserves theoriginal facial features (e.g., eye details) of the original image, userimage 900 of FIG. 9, but also appears to include shadows (to varyingdegrees and in various colors, if appropriate) casted by a pair ofglasses as if the user were actually wearing the glasses. For example,the pixel values of the shadow transform, processed inverted differenceimage 2200 of FIG. 22, are scaled down to values from 0 to 1 and thenmultiplied with the RGB channels of corresponding pixels of the originalimage, user image 900 of FIG. 9. Shadowed image 2300 comprises theoriginal image of the user's face, user image 900 of FIG. 9, with theadded glasses shadows 2304 of the shadow transform, processed inverteddifference image 2200 of FIG. 22.

FIG. 24 is a diagram showing an example presentation of a shadowed imagewith a 2D glasses image overlay. In the event that a shadowed image suchas shadowed image 2300 of FIG. 23 is to be presented during a playbackor a virtual glasses try-on at a user interface, the shadowed image isdisplayed with an overlay of a 2D glasses image. In various embodiments,the 2D glasses image that is overlaid over a shadowed image matches theorientation of the user's face in that shadowed image. In the example ofFIG. 24, displayed shadowed image 2400 may comprise a portion of aglasses virtual try-on playback at a user interface. Displayed shadowedimage 2400 comprises shadowed image 2300 of FIG. 23 displayed beneath orrather overlaid with displayed 2D glasses image 2402. For example, 2Dglasses image 2402 may be generated by transforming 3D glasses model1502 of FIG. 15 using the set of extrinsic information corresponding touser image 900 of FIG. 9 and projecting the transformed 3D glasses modelonto a 2D surface of the focal plane of a camera using a set ofintrinsic information associated with the camera.

As shown in the example, glasses shadows 2404 rendered for 2D glassesimage 2402 are visible on the user's face so as to emulate the shadowsthat would be casted by a pair of sunglasses (or regular glasses) if theuser had worn them in real life. Hence, displayed shadowed image 2400not only presents a 2D image of a pair of glasses in the sameorientation as the user's face but also realistic shadows that are toappear to be casted by those glasses on the user's face to give ahigh-quality virtual glasses try-on experience. As described above,because glasses shadows 2404 was rendered using a 3D generic face modeland then applied to an actual image of a user's face, the glasses shadoweffect can be quickly realized.

While FIGS. 9 through 24 describe an example of rendering glassesshadows on only one image of a user's face, the same process may beapplied to various different images of the user's face (at differentorientations) such that a user may interact (e.g., using a cursor or atouchscreen) with a playback of images at a user interface to viewdifferent user images displayed with images of glasses and theircorresponding simulated shadows.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. (canceled)
 2. A system, comprising: a processor;and a memory coupled with the processor, wherein the memory isconfigured to provide the processor with instructions which whenexecuted cause the processor to: generate a face image corresponding toan image of a set of images based at least in part on a face model,wherein the set of images is associated with a user's face, wherein thegenerating of the face image corresponding to the image of the set ofimages comprises to transform the face model using a set of extrinsicinformation corresponding to the image; generate a face with shadowsimage corresponding to the image based at least in part on shadowscasted by a glasses model on the face model; generate a shadow transformbased at least in part on a difference determined based at least in parton the face image and the face with shadows image; generate a shadowedimage based at least in part on applying the shadow transform to theimage; and present the shadowed image including by overlaying a glassesimage associated with the glasses model over the shadowed image.
 3. Thesystem of claim 2, wherein the processor is further configured to:obtain a three-dimensional (3D) model associated with the user's face;and generate a modified 3D generic face model by modifying a 3D genericface model to match the 3D model associated with the user's face,wherein the face model comprises the modified 3D generic face model. 4.The system of claim 3, wherein the 3D model associated with the user'sface is determined from the set of images associated with the user'sface.
 5. The system of claim 2, wherein to generate the face withshadows image corresponding to the image based at least in part onshadows casted by the glasses model on the face model comprises to:transform the glasses model using the set of extrinsic informationcorresponding to the image; generate a combination of the transformedface model with the transformed glasses model; and apply a lightingmodel to the combination to cause the transformed glasses model to castshadows onto the transformed face model.
 6. The system of claim 5,wherein the lighting model is determined based at least in part on theset of images associated with the user's face.
 7. The system of claim 2,wherein to generate the shadow transform based at least in part on thedifference determined based at least in part on the face image and theface with shadows image comprises to: generate a difference image bysubtracting the face image from the face with shadows image; invertcolors associated with the difference image to obtain an inverteddifference image; and determine the inverted difference image as theshadow transform.
 8. The system of claim 7, wherein to generate theshadow transform further comprises to apply a Despeckle filter to theinverted difference image.
 9. The system of claim 7, wherein to generatethe shadow transform further comprises to apply a blur filter to theinverted difference image.
 10. The system of claim 7, wherein togenerate the shadow transform further comprises to apply a Red GreenBlue (RGB) curves filter to the inverted difference image.
 11. Thesystem of claim 7, wherein to generate the shadow transform furthercomprises to apply a Hue Saturation Value (HSV) filter adjustment to theinverted difference image.
 12. The system of claim 2, wherein togenerate the shadowed image based at least in part on applying theshadow transform to the image comprises multiplying the image with theshadow transform.
 13. The system of claim 2, wherein the glasses modelis associated with a pair of glasses selected by a user.
 14. A method,comprising: generating a face image corresponding to an image of a setof images based at least in part on a face model, wherein the set ofimages is associated with a user's face, wherein the generating of theface image corresponding to the image of the set of images comprisestransforming the face model using a set of extrinsic informationcorresponding to the image; generating a face with shadows imagecorresponding to the image based at least in part on shadows casted by aglasses model on the face model; generating, using a processor, a shadowtransform based at least in part on a difference determined based atleast in part on the face image and the face with shadows image;generating a shadowed image based at least in part on applying theshadow transform to the image; and presenting the shadowed imageincluding by overlaying a glasses image associated with the glassesmodel over the shadowed image.
 15. The method of claim 14, furthercomprising: obtaining a three-dimensional (3D) model associated with theuser's face; and generating a modified 3D generic face model bymodifying a 3D generic face model to match the 3D model associated withthe user's face, wherein the face model comprises the modified 3Dgeneric face model.
 16. The method of claim 15, wherein the 3D modelassociated with the user's face is determined from the set of imagesassociated with the user's face.
 17. The method of claim 14, whereingenerating the face with shadows image corresponding to the image basedat least in part on shadows casted by the glasses model on the facemodel comprises: transforming the glasses model using the set ofextrinsic information corresponding to the image; generating acombination of the transformed face model with the transformed glassesmodel; and applying a lighting model to the combination to cause thetransformed glasses model to cast shadows onto the transformed facemodel.
 18. The method of claim 14, wherein generating the shadowtransform based at least in part on the difference determined based atleast in part on the face image and the face with shadows imagecomprises: generating a difference image by subtracting the face imagefrom the face with shadows image; inverting colors associated with thedifference image to obtain an inverted difference image; and determiningthe inverted difference image as the shadow transform.
 19. A computerprogram product, the computer program product being embodied in anon-transitory computer-readable storage medium and comprising computerinstructions for: generating a face image corresponding to an image of aset of images based at least in part on a face model, wherein the set ofimages is associated with a user's face, wherein the generating of theface image corresponding to the image of the set of images comprisestransforming the face model using a set of extrinsic informationcorresponding to the image; generating a face with shadows imagecorresponding to the image based at least in part on shadows casted by aglasses model on the face model; generating a shadow transform based atleast in part on a difference determined based at least in part on theface image and the face with shadows image; generating a shadowed imagebased at least in part on applying the shadow transform to the image;and presenting the shadowed image including by overlaying a glassesimage associated with the glasses model over the shadowed image.